Guest tracing improvements to use `tracing` crate #844

dblnz · 2025-08-31T13:46:42Z

Description

This PR closes #723, #704 and partially addresses #318.
These changes modify the way we perform guest tracing to use the tracing crate and its macros (instrument, trace).

How it works

Guest

What makes this possible is the implementation of the Subscriber trait in the hyperlight-guest-tracing crate. By implementing it, we can now handle the capturing of spans and events and choose how to store them and when to export them to the host.

The GuestSubscriber type that implements Subscriber keeps an internal TraceState that holds all the needed information.
Whenever a new span is created, entered or exited, a callback on the subscriber is called so that we can handle the functionality. The same happens with the events also.

Each time a new span or event is added to the internal state, we check whether the buffer got full and send them to the host to process.

Host

When the host detects a VM exit from the guest, it checks whether it contains tracing information in the OutB instruction.
When tracing information is found, the host starts going through it and check against the local storage of spans.

If the spans have previously been created, just update the end timestamp (if present) and add new events (if any).
If they haven't been created, create and store them.

The spans parents are set based on the information got from the host.

TODO

The current issue is with the guest calls that end up calling back into the host.
These do not correctly set the parents of the spans created in the host to the last one created in the guest before doing the VM exit
I need to find a way to propagate the context into the guest and back whenever it is needed. But using the Opentelemetry propagators needs std support which we do not have in the guest.
There is a corner case with calculating the timestamp for the span that is open when a VM exit is done

Jaeger picture of a Guest call that calls back into the host

jprendes

I made a high level review, and what I've seen looks good :-)

src/hyperlight_host/src/hypervisor/hyperv_windows.rs

src/hyperlight_host/src/hypervisor/hyperv_linux.rs

Cargo.toml

src/hyperlight_common/src/outb.rs

src/hyperlight_host/src/hypervisor/hyperv_linux.rs

src/hyperlight_host/src/hypervisor/hyperv_windows.rs

src/hyperlight_host/src/hypervisor/kvm.rs

ludfjig

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

Also, is there any possibility that we don't flush spans/records after exiting the guest, and that some end up not being emitted?

Another thing to consider is log crate vs tracing crate. Should we ditch one? Or is there some mechanism that allows regular logs to be consumed by tracing crate: And should we expose any of these in guest C-api?

Also is the tracing buffer sizes configurable? Maybe it should be if it isn't, so users can tweak it in case it affects performance.

src/hyperlight_common/Cargo.toml

src/hyperlight_guest/src/exit.rs

src/hyperlight_guest/src/guest_handle/host_comm.rs

src/hyperlight_guest_tracing/src/invariant_tsc.rs

jsturtevant · 2025-09-11T23:17:46Z

I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

+1

dblnz · 2025-09-12T15:27:26Z

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

Ok, I can do that.

Also, is there any possibility that we don't flush spans/records after exiting the guest, and that some end up not being emitted?

Hmm, in my limited testing I haven't seen this case, but I wouldn't exclude the possibility.
One scenario that comes to mind is cancellation, where the vCPU forced to stop, so no out instruction to deliver the info.
Any ideas how we can treat these scenarios?

Another thing to consider is log crate vs tracing crate. Should we ditch one? Or is there some mechanism that allows regular logs to be consumed by tracing crate: And should we expose any of these in guest C-api?

I am not sure about the best approach is.
The solution I added for guest spans/events only works with opentelemetry Subscribers on the host, so if no opentelemetry subscribers, the they won't be captured (haven't tried it yet, but this is how it should work)

Also is the tracing buffer sizes configurable? Maybe it should be if it isn't, so users can tweak it in case it affects performance.

The tracing buffer is compile time configurable, which I agree is not ideal for customers.
But I needed a fixed size buffer so that I could reliably give the pointer to the host to access memory (current approach, I don't know if it is the best, but it is certainly faster than using the input/output buffer which copies data twice. This way we only copy once, on the host).

dblnz · 2025-09-30T15:57:55Z

First round review looks good to me. I am curious what the performance looks like with tracing vs without, maybe we could add a benchmark or something for this?

I've run some benchmarks locally and here are the results.
Relative to the work done in the dummy guest, the tracing logic could look like it takes a lot, but we need a realistic guest scenario to correctly assess how relevant the numbers are.

Runtime	Strategy	Flavour	RPS	p50 (s)	p95 (s)	p99 (s)	p99.99 (s)	Peak RSS (MB)
hyperlight-dummy	new		69.96	0.7065	1.1569	1.2509	1.3412	12.79
hyperlight-dummy-tracing	new	nolog	70.92	0.6728	1.1247	1.2065	1.3029	20.21
hyperlight-dummy-tracing	new	log	70.14	0.7354	1.0132	1.0953	1.1922	18.54
hyperlight-dummy	reload		93506.17	0.0004	0.0011	0.0018	0.0067	15.00
hyperlight-dummy-tracing	reload	nolog	56152.01	0.0006	0.0023	0.0049	0.0575	23.42
hyperlight-dummy-tracing	reload	log	38585.95	0.0009	0.0035	0.0069	0.0507	21.90
hyperlight-dummy	reuse		80821.05	0.0005	0.0016	0.0026	0.0065	14.85
hyperlight-dummy-tracing	reuse	nolog	54745.61	0.0007	0.0020	0.0049	0.0581	23.28
hyperlight-dummy-tracing	reuse	log	41301.24	0.0009	0.0028	0.0056	0.1097	21.24

Runtimes:

hyperlight-dummy - Hyperlight Sandbox running a dummy guest (no TracingProvider)
hyperlight-dummy-tracing - Hyperlight Sandbox running a dummy guest with trace_guest enabled on both guest and host and a TracingProvider instantiated
- nolog - the max log level is none, which means the guest doesn't send tracing info, but the Host logic to handle them is enabled
- log - the max log level is trace for the guest, which makes the guest send tracing info and the host to handle them

Some thoughts:

This feature started as a development phase improvement to help figure out where we have bottlenecks on the guest.
This is a huge improvement over the previous solution we had.
At this point, I do not think this should be used in production by default because the performance is not ideal. It could be useful on an error path, when something doesn't work as expected, it could be turned on
These numbers do not perfectly reflect only the tracing logic comparison because there are multiple variables such as: the communication with the Jaeger Collector to send the tracing data (when the TracingProvider is installed).
Some user feedback would be perfect
This would benefit from other generic performance improvements, such as (sharing memory with the guest in such a way that doesn't imply copying memory - guest immutable references).
I think we should first discuss where we stand in terms of these big changes in Hyperlight, which may also positively impact the tracing performance.

ludfjig

This looks good to me in general, but I think maybe we should consider what tunables we need

src/hyperlight_guest_bin/src/lib.rs

docs/hyperlight-metrics-logs-and-traces.md

dblnz · 2025-10-15T07:53:15Z

This looks good to me in general, but I think maybe we should consider what tunables we need

I see this version using opentelemetry as an incremental step, after which additional tunables can be added to improve usability.
Unless there is any fundamental issue with this approach, I think this can be merged and improved after.

Improvements:

Switch to using a circular buffer for traces/events to avoid context switches
Provide a mechanism on the guest to enable/disable the tracing support, configure the memory used and log level at runtime:

jsturtevant

Really like the idea of using the industry standard solution for tracing!

src/hyperlight_guest_bin/src/lib.rs

src/hyperlight_guest_tracing/src/visitor.rs

src/hyperlight_host/src/hypervisor/arch/mod.rs

src/hyperlight_host/src/sandbox/initialized_multi_use.rs

simongdavies

I've made a couple of comments , I don't think that any of these are things that need to be addressed here but we should consider as future enhancements

src/hyperlight_guest_bin/src/lib.rs

docs/hyperlight-metrics-logs-and-traces.md

src/hyperlight_guest_tracing/src/lib.rs

ludfjig

I'm a big fan of this! Btw what happens when buffer in guest is full and an event/span comes in? Also question: is the parent=Span::current required inside all the
#[instrument(skip_all, parent = Span::current(), level= "Trace")]? Or is it the default

src/hyperlight_common/src/outb.rs

dblnz · 2025-10-22T21:28:44Z

The buffer length is verified every time after a new event/span is pushed. So if it got full, it is emptied afterwards.

The ´parent = Span::current()´ is not needed. I am planning to change that afterwards with some other small things.

- This feature is not used separate from the mem_profile - All the unwind logic is now gated by mem_profile Signed-off-by: Doru Blânzeanu <[email protected]>

- The guest side does not use this type of OutBAction - The stack unwinding is done either way when the mem_profile feature is enabled Signed-off-by: Doru Blânzeanu <[email protected]>

- This helps with keeping code separate and easily gating it out Signed-off-by: Doru Blânzeanu <[email protected]>

- This steps cleans up codebase for the new way of tracing guests - The current method involves custom macros and logic that are not the best for maintainability Signed-off-by: Doru Blânzeanu <[email protected]>

- Define a separate struct that holds the functionality related to memory profiling of the guest Signed-off-by: Doru Blânzeanu <[email protected]>

- Rename TraceInfo to reflect only being used by mem_profile Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- Adds a type that implements the Subscriber trait of the tracing_core crate that allows the type to be set as the global Subscriber of the crate - This way we can handle the adding of new spans and events and store them where/how we want Signed-off-by: Doru Blânzeanu <[email protected]>

- implement add_span and event methods that store the info and report it to the host when the buffer gets full Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- Parse the spans and events coming from the guest and create corresponding spans and events from the host that mimics a single call from host - Create a `TraceContext` that handles a call into a guest Signed-off-by: Doru Blânzeanu <[email protected]>

- conditionally handle logs either through tracing or the dedicated VM exit based on whether tracing is initialized on the guest - modify `test_with_small_stack_and_heap` to 18kB because the `#[intrument]` attributes use more stack. Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

dblnz requested review from danbugs, devigned, jprendes, ludfjig, marosset, simongdavies and syntactically as code owners August 31, 2025 13:46

dblnz added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Aug 31, 2025

dblnz changed the title ~~Tracing improvements~~ Guest tracing improvements to use tracing crate Aug 31, 2025

dblnz marked this pull request as draft August 31, 2025 13:48

jprendes reviewed Sep 1, 2025

View reviewed changes

dblnz force-pushed the tracing-improvements branch 2 times, most recently from d4327a8 to ccaa14e Compare September 10, 2025 22:45

dblnz marked this pull request as ready for review September 10, 2025 22:45

dblnz force-pushed the tracing-improvements branch from ccaa14e to cceb069 Compare September 10, 2025 22:47

ludfjig reviewed Sep 11, 2025

View reviewed changes

src/hyperlight_common/Cargo.toml Show resolved Hide resolved

src/hyperlight_guest/src/exit.rs Outdated Show resolved Hide resolved

src/hyperlight_guest/src/guest_handle/host_comm.rs Show resolved Hide resolved

src/hyperlight_guest_tracing/src/invariant_tsc.rs Show resolved Hide resolved

dblnz force-pushed the tracing-improvements branch 6 times, most recently from 77bbba5 to 6d10d2e Compare September 18, 2025 21:27

dblnz force-pushed the tracing-improvements branch from 6d10d2e to 8ff5a1e Compare September 30, 2025 15:25

dblnz force-pushed the tracing-improvements branch from 8ff5a1e to f00d63e Compare October 1, 2025 09:55

ludfjig reviewed Oct 1, 2025

View reviewed changes

src/hyperlight_guest_bin/src/lib.rs Outdated Show resolved Hide resolved

docs/hyperlight-metrics-logs-and-traces.md Show resolved Hide resolved

dblnz force-pushed the tracing-improvements branch from f00d63e to 148470a Compare October 6, 2025 15:44

dblnz force-pushed the tracing-improvements branch from 148470a to 908f482 Compare October 13, 2025 16:00

jsturtevant reviewed Oct 16, 2025

View reviewed changes

simongdavies previously approved these changes Oct 20, 2025

View reviewed changes

src/hyperlight_guest_bin/src/lib.rs Show resolved Hide resolved

docs/hyperlight-metrics-logs-and-traces.md Show resolved Hide resolved

src/hyperlight_guest_tracing/src/lib.rs Show resolved Hide resolved

dblnz dismissed simongdavies’s stale review via 0e6f377 October 21, 2025 10:50

dblnz force-pushed the tracing-improvements branch from 6468526 to 0e6f377 Compare October 21, 2025 10:50

dblnz mentioned this pull request Oct 21, 2025

Address guest tracing improvements #964

Open

dblnz force-pushed the tracing-improvements branch 2 times, most recently from e1c72e2 to dfa8cb0 Compare October 22, 2025 10:02

ludfjig previously approved these changes Oct 22, 2025

View reviewed changes

src/hyperlight_common/src/outb.rs Outdated Show resolved Hide resolved

dblnz added 14 commits October 23, 2025 12:20

[trace-guest] remove unwind_guest feature

3c5e38d

- This feature is not used separate from the mem_profile - All the unwind logic is now gated by mem_profile Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-host] remove unused OutBAction::TraceRecordStack

b50d37a

- The guest side does not use this type of OutBAction - The stack unwinding is done either way when the mem_profile feature is enabled Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-host] move trace related logic to separate module

f0f7825

- This helps with keeping code separate and easily gating it out Signed-off-by: Doru Blânzeanu <[email protected]>

[trace] remove old tracing functionality

0b8da89

- This steps cleans up codebase for the new way of tracing guests - The current method involves custom macros and logic that are not the best for maintainability Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-host] refactor mem_profile logic

1837342

- Define a separate struct that holds the functionality related to memory profiling of the guest Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-host] move mem_profile logic to different file

e734010

- Rename TraceInfo to reflect only being used by mem_profile Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-guest] move invariant_tsc logic to separate module

e4a0785

Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-guest] add span and event storage logic

4a91054

- implement add_span and event methods that store the info and report it to the host when the buffer gets full Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-guest] update outb instructions to include trace info

1e3807c

Signed-off-by: Doru Blânzeanu <[email protected]>

[trace-guest] add spans in simpleguest sample

4ac0278

Signed-off-by: Doru Blânzeanu <[email protected]>

update tracing docs

1328218

Signed-off-by: Doru Blânzeanu <[email protected]>

dblnz dismissed ludfjig’s stale review via 1328218 October 23, 2025 09:28

dblnz force-pushed the tracing-improvements branch from dfa8cb0 to 1328218 Compare October 23, 2025 09:28

simongdavies approved these changes Oct 23, 2025

View reviewed changes

dblnz merged commit 1328218 into hyperlight-dev:main Oct 23, 2025
49 checks passed

This was referenced Oct 23, 2025

Update guest traces to standard format #704

Closed

Investigate making trace spans flow into the hyperlight guest #318

Closed

Guest tracing improvements to use tracing crate #844

Guest tracing improvements to use tracing crate #844

Conversation

dblnz commented Aug 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How it works

Guest

Host

TODO

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ludfjig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsturtevant commented Sep 11, 2025

Uh oh!

dblnz commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dblnz commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dblnz commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsturtevant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simongdavies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dblnz commented Oct 22, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Guest tracing improvements to use `tracing` crate #844

Guest tracing improvements to use `tracing` crate #844

dblnz commented Aug 31, 2025 •

edited

Loading

ludfjig left a comment •

edited

Loading

dblnz commented Sep 12, 2025 •

edited

Loading

dblnz commented Sep 30, 2025 •

edited

Loading

dblnz commented Oct 15, 2025 •

edited

Loading